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Overview 

<*  What  is  Collaborative  Cognition? 

A  Focus  on  Measurement 

❖  Assessing  Collaborative 
Performance  &  Cognition 

❖  Toward  Diagnosis  of  Collaborative 
Performance 

❖  Conclusions 


What  is 
Collaborative 
Cognition? 


Experimental  Context 

CERTT  (Cognitive  Engineering  Research  on 

Team  Tasks)  Lab 

A  Synthetic  Task  Environment  for  the  Study  of  Coiiaborative 

gnition 


Five  Participant  Consoies 


Experimenter  Consoie 
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Collaborative  Cognition  Framework 


Collective  level 


Holistic  Level 


January  2003 


Team  Process 
Behaviors 


Team 

Knowledge 
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Individual 

knowledge 


Taskwork  &  Teamwork 
Knowledge 

Long-term 

Fleeting 
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Defining  Collaborative 

Cognition 

»  It  is  more  than  the  sum 
of  the  cognition  of 
individual  team 
members. 

»  It  emerges  from  the 
interplay  of  the 
individual  cognition  of 
each  team  member  and 
team  process  behaviors 
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A  Focus  on 
Measurement 


Why  Measurement? 

Assessment  of  collaborative  performance 
or  effectiveness  (criterion  or  dependent 
measures)  often  taken  for  granted. 

Outcome  measures  of  collaborative 
performance  do  not  reveal  why 
performance  is  effective  or  ineffective. 

Process  measures  of  collaborative 
behavior  are  often  subjective  and  lack 
reliability  and  validity. 
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Why  Measurement? 
(continued) 

Collaborative  cognition  is  assumed  to 
contribute  to  collaborative  performance, 
and  especially  for  growing  number  of 
cognitive  tasks. 

Understanding  the  team  cognition  behind 
team  performance  should  inform 
interventions  (design,  training,  selection) 
to  improve  that  performance. 

- ( 
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Measurement  Limitations 

Measures  tend  to  assume  homogeneous  groups 

Measures  tend  to  target  collective  level 

Aggregation  methods  are  limited 

Measures  are  needed  that  target  the  more  dynamic 
and  fleeting  knowledge 

Measures  are  needed  that  target  different  types  of 
long-term  collaborative  knowledge 

A  broader  range  of  knowledge  elicitation  methods  is 
needed 

A  need  for  streamlined  and  embedded  measures 
Newly  developed  measures  require  validation 


Assessing 
Collaborative 
Performance 
and  Cognition 

CKM  workshop 
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The  "Apples  and  Oranges 

Problem 


Measures  to  assess  collaborative  knowledge 
often  assume  knowledge  homogeneity  among 
group  members. 


►Shared  knowledge  = 
similar  knowledge 

►Accuracy  is  relative  to 
single  referent  - 


Person  A  Person  B 


Referent 
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The  Groups  We  Study 

Consist  of 

"Apples  and  Oranges" 

iT 


Airport  Incident  Command  Center 


Telemedicine 
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Shared  Knowledge 


Shared  =  Common  and  Complementary 
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An  Approach  to  the  Apples 
and  Oranges  Problem 

Measures  of  team 
knowledge  with 
heterogeneous 
accuracy  metrics 
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Experimental  Context 

Five  studies:  Two  different  3-person 
tasks:  UAV  (Uninhabited  Air  Vehicle)  and 
Navy  helicopter  rescue-and-relief 

Procedure:  Training,  several  missions, 
knowledge  measurement  sessions 

Manipulate:  co-located  vs.  distributed 
environments,  training  regime, 
knowledge  sharing  capabilities,  workload 

m. 
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Experimental  Context 

MEASURES 

❖  Team  performance:  composite  measure 

❖  Team  process:  observer  ratings  and 
critical  incident  checklist 

❖  Other:  Communication  (flow  and  audio 
records),  video,  computer  events, 
leadership,  demographic  questions, 
working  memory,  situation  awareness 

♦>  Taskwork  &.  Teamwork  Knowledge 


Team  Performance  Score 


Long-term  Taskwork 

Knowledge 

Factual  Tests 

The  camera  settings  are  determined  by  a) 
altitude,  b)  airspeed,  c)  light  conditions,  d)  all  of 
the  above. 


Psychological  scaling 

How  related  is  airspeed  to  restricted  operating 
zone? 
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Long-term  Teamwork 

Knowledge 

Given  a  specific  task  scenario,  who 
passes  what  information  to  whom? 


Teamwork  Checklist 

AVO  gives  airspeed  info  to  PLO 
.DEMPC  gives  waypoint  restrictions  to  AVO 
.PLO  gives  current  position  to  AVO 


AVO=  Air  Vehicle  Operator 
PLO  =  Payload  Operator 
DEMPC  =  Navigator  _ 
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Traditional  Accuracy 

Metrics 


Team  Referent 


.50 


Team  Member: 

Air  Vehicle  Operator 


50%  ACCURACY 
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Heterogeneous  Accuracy 

Metrics 


AVO=  Air  Vehicle  Operator 
PLO  =  Payload  Operator 
DEMPC  =  Navigator 


Overall:  50% 
Positional:  100% 
Interpositional:  17% 


♦>  ♦>  ♦> 


Results  Across  Studies 


Taskwork  knowledge  is 
predictive  of  team  performance 

But... 

True  for  psychological  scaling, 
not  factual  tests 

Timing  of  knowledge  test  is 
critical 

CKM  workshop 
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Knowledge  Profiles  of  Two  Tasks 


Knowledge 
profile 

characterizing! 
effective 
teams 
depends  on 
task  (UAV  vs. 
Navy) 


Knowledge  Profile 

Common 

(UAV) 

Distributed 

(Navy 

helicopter) 

0 

0 

0 
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Knowledge  Profiles  of  Two 

Tasks 


Common  Complementary 


IUAV  Task  Navy  Helicopter  Task 

Command-and-Control  Planning  and  Execution 
Interdependent  Less  interdependent 

Knowledge  sharing  Face-to-Face 
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A  Cross-Training  Study  in 

Retrospect 

Examined  effects  cross-training  vs.  other  training 
regimes  on  collaborative  performance  and  cognition 

Unlike  previous  studies,  cross-training  had  no 
performance  benefit 

Cross-training,  did  increase  interpositional  taskwork 
and  teamwork  knowledge 

Perhaps  knowledge  profile  for  that  task 
(specialization)  was  at  odds  with  cross-training 

Demonstrates  benefits  of  assessing  collaborative 


January  2003 


However... 

The  descriptive  information  associated 
with  cognitive  assessment  is  not 
sufficiently  diagnostic 


Symptoms  vs.  Diagnoses? 

Expert  chess  players  can  remember  many  more 

meaningful  chess  positions  than  chess  novices 

Experienced  fighter  pilots  and  undergraduate 

pilot  trainees  organize  flight  maneuver 

concepts  differently 

Good  UAV  teams  have  interpositional 

knowledge 

Effective  teams  communicate  less  than 
ineffective  ones 


Toward 
Diagnosis  of 
Collaborative 

Performance 
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To  Move  From  Assessment 

to  Diagnosis  .... 

Need  to  connect  clusters  of  symptoms  to 
diagnosis  of  team  dysfunction  or  excellence 


Inefficient  communication  and  low  teamwork 
knowledge  poor  team  situation  awareness 


Poor  positional  taskwork  knowledge  and 
coordination  failures  faulty  mental  model 
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To  Move  From  Assessment 

to  Diagnosis  .... 

Also,  in  operational  environments  diagnosis 
is  valuable  to  the  extent  that  it  is... 


Conducted  in  real  time  with  task 
performance  (i.e.,  task-embedded, 
automated  measures) 

Or  better  yet  ...prior  to  task  performance 
(based  on  performance  precursors) 

Leading  indicator 
Resident  pathogens 
Non-routine  events 
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Communication  as  a  Window 
to  Coiiaborative  Cognition 


►  Observable;  Think  aloud  "in  the  wild" 

►  Reflects  collaborative  cognition  at  the 
holistic  level;  is  collaborative  cognition 

►  Embedded  in  the  task 

►  But.. .labor  intensive  transcription,  coding, 
and  interpretation 

►  Exploit  its  richness  by  automating 
analyses 
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Our  Approach  to 
Communication  Anaiysis 


Communication  Flow  Analysis 

Content  Analysis  Using  Latent 
Semantic  Analysis 
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Analyzing  Flow:  CERTT  Lab 

ComLog  Data 

LJi 


Team  members  use  push-to-talk  intercom 
buttons  to  communicate.  At  regular  intervals 
speaker  and  listener  identities  are  logged 
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Analyzing  Flow:  ProNet-- 
Procedural  Networks 

Nodes  define  events  that  occur  in  a  sequence 

An  Example  from  UAV  study:  6  nodes:  Abeg,  Aend, 
Pbeg,  Pend,  Dbeg,  Dend 

ProNet:  Find  representative  event  sequences 


Quantitative:  Chain  lengths— >Performance 

Mission  2:  =  .509,  F(2,  8)  =  4.144,  p  =  .058 

Mission  3:  =  .275,  F(l,  9)  =  3.415,  p  =  .098 

Mission  5:  R^  =  .628,  F(2,  8)  =  5.074,  p  =  .051 
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Analyzing  Flow:  ProNet-- 
Procedural  Networks 


AVO=  Air  Vehicle  Operator 
PLO  =  Payload  Operator 
DEMPC  =  Navigator 


♦>  ♦>  ♦> 


Analyzing  Flow:  Other 

Approaches 

Measure  of  speaker  dominance 
Deviations  from  ideal  flow 
Clustering  model-based  patterns 
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Content  Analysis  with 
Latent  Semantic  Analysis  (LSA) 


Landauer,  Foltz,  &  Laham,  1998 


❖  A  tool  for  measuring  cognitive  artifacts  based 
on  semantic  information. 

❖  Provides  measures  of  the  semantic 
relatedness,  quality,  and  quantity  of 
information  contained  in  discourse. 

*>  Automatic  and  fast. 

<*  We  can  derive  the  meaning  of  words  through 
analyses  of  large  corpora. 
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Content  Analysis  with 
Latent  Semantic  Analysis  (LSA) 

(continued) 

Large  constraint  satisfaction  problem  of 
estimating  the  meaning  of  many  passages 
based  on  their  contained  words  (like  factor 
analysis) 

Method  represents  units  of  text  (words, 
sentences,  discourse,  essays)  as  vectors  in  a 
high  dimensional  semantic  space  based  on 
correlations  of  usage  across  text  contexts 

Compute  degree  of  semantic  similarity 
between  any  two  units  of  text 
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Content  Analysis  with 
Latent  Semantic  Analysis  (LSA) 

An  Example  from  UAV  Study  1 

67  Transcripts  from  missions  1-7 

XML  tagged  with  speaker  and  iistener  information 

~2700  minutes  of  spoken  diaiogue 

20,545  separate  utterances  (turns) 

232,000  words  (660  k  bytes  of  text) 

Semantic  Space:  22,802  documents 

Utterances  from  diaiogues 

Training  materiai 

Interviews  with  domain  experts 
Derived  severai  statisticai  measures  of  the  quaiity  of 
each  transcript - 


Content  Analysis  with 
Latent  Semantic  Analysis  (LSA) 


#  Team  1  Mission  3 
Score:  750 

0  Team  3  Mission  4 

Score:  620 

Team  8  Mission  6 
0  Score  560 

Team  7  Mission  3 

Score  580 

8  Mission  3 
???? 

#  Team  6  Mission  3 
Score  490 

Team  5  Mission  4 

Score  460 

LSA-based  communication  score 
predicts  performance  (r  =.79). 
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other  Significant  Variables 

❖Variance  of  scores  of  similar  dialogues  r=-.58 
❖Vector  length,  r=-.35 

❖  Number  of  words,  r=-.34 

❖  Zipf  R2,  r=-.47 

❖  Percent  non-function  words,  r=.34 

❖  .... 

❖  Five  factor  RMMR  model:  r=.76 

Conclusion:  We  can  accurately  predict  team 
performance  from  dialogues  as  a  whole. 
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Analyzing  Content: 
Other  Approaches 

❖  Automatic  transcript  coding 

❖  Coherence  in  team  dialogue 

❖  Measures  of  individual 
contributions 
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Implications  of  Communication 

Work 

Collaborative  cognition  is  revealed  through 
discourse 

Measurement  of  the  team  as  a  whole,  as  well 
as  individuals 

Techniques  move  toward  automated  analyses 
of  the  content  of  team  dialogues 

Avoids  tedious  hand  coding,  keeps  high 
reliability 

Automation  will  allow  for  task-embedded 
naeasures  that  assess^araci  diagnose  in  real-time 


Conclusions 
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Summary 

Understanding  collaborative  cognition 
is  critical  for  diagnosis  of  team 
dysfunction  or  excellence  and  later 
intervention 

Assessment  is  only  a  first  step 

Diagnostic  information  is  needed 

In  operational  environments  diagnosis 
needs  to  be  task-embedded, 
automatic,  and  forward-looking 
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Implications  &  Applications 

Suggestions  for  aiding  collaborative 
cognition  through  training  or 
technology. 

Selecting/composing  teams  for 
optimal  collaboration 

On-line  monitoring  of  collaborative 
cognition  in  high-risk  environments 
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Questions  or  Comments? 


Contact 
Nancy  J.  Cooke 
Arizona  State  University 
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